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Abstract 

Background: Updating clinical practice guidelines (CPGs) is a crucial process for maintaining the validity of 
recommendations. Methodological handbooks should provide guidance on both developing and updating CPGs. 
However, little is known about the updating guidance provided by these handbooks. 

Methods: We conducted a systematic review to identify and describe the updating guidance provided by CPG 
methodological handbooks and included handbooks that provide updating guidance for CPGs. We searched in the 
Guidelines International Network library, US National Guidelines Clearinghouse and MEDLINE (PubMed) from 1966 
to September 2013. Two authors independently selected the handbooks and extracted the data. We used 
descriptive statistics to analyze the extracted data and conducted a narrative synthesis. 

Results: We included 35 handbooks. Most handbooks (97.1%) focus mainly on developing CPGs, including variable 
degrees of information about updating. Guidance on identifying new evidence and the methodology of assessing 
the need for an update is described in 1 1 (31.4%) and eight handbooks (22.8%), respectively. The period of time 
between two updates is described in 25 handbooks (71.4%), two to three years being the most frequent (40.0%). 
The majority of handbooks do not provide guidance for the literature search, evidence selection, assessment, 
synthesis, and external review of the updating process. 

Conclusions: Guidance for updating CPGs is poorly described in methodological handbooks. This guidance should 
be more rigorous and explicit. This could lead to a more optimal updating process, and, ultimately to valid 
trustworthy guidelines. 

Keywords: Clinical practice guidelines. Evidence-based medicine. Handbooks, Methodology, Systematic review 



Background 

Clinical practice guidelines (CPGs) intend to patient care 
by providing recommendations about the benefits and 
downsides of best practice in healthcare [1]. If adequately 
implemented, CPGs have the potential of reducing vari- 
ability and translating scientific research into clinical prac- 
tice and consequentiy improve the quality and safety of 
healthcare [2-4]. 

However, scientific knowledge is in constant change; 
therefore CPGs need to be updated regularly to maintain 
validity [5]. The obsolescence of a CPG might occur be- 
cause of new scientific research, including the develop- 
ment of new technologies in treatment and diagnosis 
alternatives, economic differences, or changes in values 
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and preferences [6,7]. Generally, an updating process con- 
sists of three components: the identification of new evi- 
dence, the assessment of the need to update, and the 
formulation of new or modified recommendations [5,8-11]. 
Some authors suggest that an update is generally required 
after three to five years; however, little research has been 
undertaken so far [8,12,13]. 

Several institutions responsible for developing CPGs 
drafted their own methodological handbooks including 
methodology for developing and updating their CPGs. 
Some of these handbooks are very influential and often 
used in smaller organizations [6,14]. Even though the 
methodology developed greatly over the last years, the 
quality of CPGs is lagging behind [1,15,16]. A lack of 
compliance with state of the art methodology for devel- 
oping CPGs has been found, and hence the methodo- 
logical quality of CPGs remained very similar over the 
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last two decades [17,18]. Little is known about the guid- 
ance for updating CPGs included in these handbooks 
[19,20]. Therefore, we systematically reviewed CPGs 
methodological handbooks to identify and describe the 
methodological guidance about updating. 

Methods 

Search strategy 

We conducted a systematic search in September 2013 in 
MEDLINE (via PubMed, from 1966 onwards), using a com- 
bination of free text terms (Clinical Practice Guidelines, 
Clinical Guidelines, Guidelines, Methodolog*, Handbook*). 
The search strategy is available as supplementary data 
(Additional file 1). In addition, we searched: the data- 
base of the Guidelines International Network (http:// 
www.g-i-n.net); the US National Guidelines Clearing- 
house database (http://www.guidelines.gov); and the 
website of institutions that reported to use a methodo- 
logical handbook in a previous international survey con- 
ducted by our group [12]. If necessary, we contacted 
organizations to obtain the handbooks. 

Eligibility criteria 

We included methodological handbooks that provide 
guidance on the updating process of CPGs. Handbooks 
that exclusively report methodologies for developing 
de novo guidelines were excluded. We included hand- 
books regardless of their language or publication status. 
When necessary, the handbook was translated. 

Study selection 

Two authors (RV, AJS) independently selected potential 
handbooks by reviewing titles and abstracts, and finally 
full text for a more detailed evaluation. Disagreements 
were initially resolved by consensus, and if necessary, 
with the help of a third author (PA-C). 

Data extraction 

Based on our previous experiences concerning updating, 
including an international survey [12] a systematic review 
[8] and additional relevant literature [5,6,9-11,14] we de- 
veloped, reviewed, and piloted iteratively a case report 
form (CRF). After consensus, the following items are in- 
cluded in the CRF: characteristics of the handbook and 
institution, group responsible for updating CPGs, strat- 
egy for identifying new evidence, methodology for 
assessing the need for an update, methods for the litera- 
ture search, evidence selection, evidence assessment, 
evidence synthesis, external review, and for the edition 
and dissemination of the updated CPG. The CRF can be 
made available upon request. 

Two authors (RV, AJS) extracted independently the data 
of the handbooks accepted for inclusion. Disagreements 
were initially resolved by consensus, and if necessary, with 



the help of a third author (PA-C). While extracting the 
data, we considered a strategy to be specific if the hand- 
book included a detailed methodology, enabling the reader 
to conduct the suggested strategy. We considered a non- 
specific strategy if not enough methodological guidance is 
provided to facilitate an adequate approach. 

Data analysis 

We used descriptive statistics to analyze the extracted 
data. We calculated absolute frequencies and propor- 
tions for all items. In addition, we conducted a narrative 
synthesis. Data analysis was performed using SPSS statis- 
tical software, version 18.0 (SPSS INC., Chicago, IL, 
USA). By consensus of two authors (RV, AJS), we col- 
lected relevant quotations within the themes included in 
the handbooks and provide these in the free text area. 

Results 

Handbooks selection 

We screened the titles and abstracts of 1,992 references 
(Figure 1). We selected 94 articles for full-text review. 
Thirty-eight articles were excluded because they were 
not methodological handbooks. Additionally, ten hand- 
books were excluded because they exclusively focused 
on developing de novo CPGs. We could not locate eight 
articles and one article was a summary of an included 
handbook. Two handbooks were excluded because a 
more recent version was included. Additional file 2 pro- 
vides an overview of the excluded documents. Finally, 
we included thirty-five handbooks (Additional file 3) 
[5,6,14,21-52]. 

Handbooks characteristics 

In total, 48.6% of the included handbooks are developed 
by institutions based in Europe [5,6,14,21-34] mostly 
being public institutions (57.1%) (Table 1) [5,6,14,22- 
26,28,31,35-43]. One handbook (2.9%) addresses specif- 
ically the methodology of updating CPGs [5]; the others 
(97.1%) focus mainly on developing de novo CPGs, and 
include variable degrees of information about updating 
[6,14,21-52]. Fourteen handbooks (40.0%) are published 
between 2005 and 2010 [5,21,23,26,30,32,34,39,40,43, 
44,46,48,50]. 

Updating group 

The persons responsible for updating the CPG are speci- 
fied in twelve handbooks (34.3%). Seven handbooks 
(20.0%) state that the updating group should have a simi- 
lar structure to the group that contributed to developing 
the CPG [6,14,23,30,37,44,45]. Four handbooks (11.4%) 
state that the group, responsible for updating the CPG, 
should be tailored to the new scope of the guideline 
[5,38,39,41]. 
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Figure 1 Flow chart of the screening literature process. 



Time between updates 

Twenty-five (71.4%) of the included handbooks recom- 
mend a time fi:ame between pubHshing a CPG and com- 
mencing an updating process (Table 2), with two to three 
years being the most firequently recommended (40.0%) 
[5,6,14,22,27,28,30-32,37,39,41,45,46]. Furthermore, three 
handbooks (8.6%) suggest a time firame of less than one 
year [33,34,44], and eight handbooks (22.9%) include a 
four to five year time frame [24,36,38,42,43,47-49]. 

Identification of new relevant evidence 

Eleven handbooks (31.4%) provide guidance on how to 
identify new relevant evidence. Of these eleven hand- 
books, six (17.1%) suggest using opinions or experiences 
from experts, users, or members of the original deve- 
lopment group for identifying new relevant evidence 
[5,14,23,37,43,46]. Five handbooks (14.3%) provide guid- 
ance on conducting limited searches to identify new 
relevant evidence [5,37-39,47]. Furthermore, two hand- 
books (5.7%) propose the editorial board to have peri- 
odic meetings to discuss topics with experts [32,33]. 
One handbook (2.9%) suggests collecting alerts to iden- 
tify newly published articles [5]. Externally reviewing the 
CPG by experts, who were not involved in developing 
the CPGs, is recommended by one handbook (2.9%) 
[47]. Two other handbooks (5.7%) provide a 'non-spe- 
cific strategy' and only emphasize the importance of 
identifying new relevant evidence (Table 2) [23,28]. 
Figure 2 shows examples of relevant passages included 
in the handbooks. 



Assessment of the need for an update 

The methodology of assessing the need for an update 
is described in eight handbooks (22.8%). Six of them 
(17.1%) give guidance on how to assess the importance 
and relevance of the new evidence, the disagreement be- 
tween the new evidence and current recommendations, 
and whether the new knowledge is not yet included 
[5,6,23,38,43,49]. Two handbooks (5.7%) recommend ex- 
pert judgment to assess the need for an update [38,40]. 
Producing and regularly updating evidence summaries 
and assessing the need for an update with these summar- 
ies are described in one handbook (2.9%) (Figure 2) [32]. 

Updating recommendations 

Eight handbooks (22.9%) provide guidance on what type of 
update is required in specific situations, by making a dis- 
tinction between partial or full updates (Table 2) [5,6,14, 
33,37,38,43,44]. 

Guidance for conducting a literature search strategy is 
included in seventeen handbooks (48.6%). Eight of them 
(22.8%) include guidance to adjust the original search 
strategy [5,6,14,24,26,27,37,43]. Four handbooks (11.4%) 
provide guidance on what kind of evidence to search for, 
including evidence based guidelines, health technology 
assessments, systematic reviews, and randomized con- 
trolled trials [14,27,38,41]. Two handbooks (5.7%) recom- 
mend to include a medical librarian or research officer in 
the team to conduct the literature searches [41,48]. Using 
multiple databases, e.g., MEDLINE and Cochrane Library, 
in the search strategy is recommended by two handbooks 
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Table 1 Characteristics of institutions and handbooks 



Institution characteristics 





n 


(%) 


Continent 






Europe 


17 


48.6 


North America 


12 


34.3 


Oceania 


4 


114 


nternational 


2 


5.7 


Type of organization 






Public institution 


20 


57.1 


Scientific society 


9 


25.7 


Private organism 


3 


8.6 


Other (Federal institute, NGO) 


3 


8.6 


Number of years developing guidelines 






<10 years 


10 


28.6 


10-20 years 


19 


54.3 


>20 years 


6 


17.1 


Number of guidelines published 






<5 per year 


22 


62.9 


>5 per year 


8 


22.9 


Unl<nown 


5 


14.3 


Handbook characteristics 


Type of handbook 






Development CPG handbook 


34 


97.1 


Update CPG handbook 


1 


2.9 


Publication date 






Before the year 2004 


8 


22.9 


Between 2005 - 2010 


14 


40.0 


Between 201 1 - 2013 


8 


22.9 


Unknown 


5 


14.3 



(5.7%) [41,43]. Furthermore, six handbooks (17.1%) suggest 
using the original strategy used for the development of the 
original guideline (Table 2, Figure 2) [23,28,34,40,44,50]. 

Eleven handbooks (31.4%) provide guidance for select- 
ing adequate evidence in the updating process. Three 
handbooks (8.6%) provide specific guidance on how to dis- 
card irrelevant information [5,14,44]. Eight handbooks 
(22.9%) refer the reader to the development process for 
guidance on evidence selection [6,27,28,34,37,38,48,50]. 

Guidance for evidence assessment is provided in thir- 
teen handbooks (37.1%). The assessment of the available 
evidence on the consistency, directness, validity or reliabil- 
ity is described in four handbooks (11.4%) [14,37,43,48]. 
Using critical appraisal frameworks, like OstFLCritica, is 
recommended in one handbook (2.9%) (Figure 2) [5]. 
Eight handbooks (22.9%) recommend the same original 
development strategy [6,23,27,28,34,38,44,50]. 



Table 2 Guidance reported in the included handbooks 



Group responsible for updating CPG 





n 


(%) 


Are the participants in the updating group specified? 






Yes 


12 


343 


No 


23 


65.7 


What members do the updating group consist of? 






Similar to the development team 


7 


20.0 


Updating group specifically defined 


4 


114 


Not defined 


24 


68.5 


Identification of new evidence 


Time frame for updating 






<1 year 


3 


8.5 


2-3 years 


14 


40.0 


4-5 years 


8 


22.9 


No specific time frame indicated 


10 


28.5 


Identification of new evidence 






Specific strategy 


9 


25.7 


Non specific strategy 


2 


5.7 


Not defined 


24 


68.5 


Assessment of the need for an update 


Assessment of the need for an update 






Specific strategy 


8 


22.8 


Not defined 


27 


77.1 


Updating strategy 


Distinction between different updates (partial / full) 






Yes 


8 


22.9 


No 


27 


77.1 


Literature search 






Specific strategy 


11 


314 


Similar to the development process 


6 


17.1 


No strategy defined 


18 


514 


Evidence selection 






Specific strategy 


3 


8.5 


Similar to the development process 


8 


22.9 


Not defined 


24 


68.5 


Evidence assessment 






Specific strategy 


5 


14.3 


Similar to the development process 


8 


22.9 


Not defined 


22 


62.9 


Evidence synthesis 






Specific strategy 


3 


8.5 


Similar to the development process 


5 


14.3 


Not defined 


27 


77.1 
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Table 2 Guidance reported in the included handbooks 

(Continued) 



External review 






Specific strategy 


5 


14.3 


Similar to development process 


6 


17.1 


Non specific strategy 


2 


5.7 


Not defined 


22 


62.9 


Edition and dissemination 


Indication of clianges 






Specific strategy 


5 


143 


Not defined 


30 


85.7 


Dissemination of tlie updated CPG 






Specific strategy 


3 


8.6 


Not defined 


32 


914 



Similarly, guidance for the evidence synthesis is de- 
scribed in eight handbooks (22.9%). Three handbooks 
(8.6%) recommend producing evidence tables including 
the characteristics of included studies, quality of ran- 
domized trials, results for continuous outcomes, and re- 
sults for dichotomous outcomes [14,43,48]. Moreover, 
five handbooks (14.3%) direct the reader to the section 
with guidance for evidence synthesis used for developing 
de novo CPGs [5,6,34,44,50]. 



Guidance for an external review of the updated CPG is 
described in thirteen handbooks (37.1%). Five handbooks 
(14.3%) describe the process of external reviewing the up- 
dated CPG by multiple external reviewers [37,43,45,47,48]. 
Furthermore, two handbooks (5.7%) provides 'non-specific 
guidance' for conducting an external review of the updated 
CPG [28,38]. Sbc handbooks (17.1%) refer to the guidance 
described in the section of developing de novo CPGs 
[5,6,27,34,44,50]. 

Edition and dissemination 

Two handbooks (5.7%) suggest to post a notification on 
the website of the institution whenever the need for 
an update is confirmed [28,29]. Five handbooks (14.3%) 
include a specific strategy for indicating the changes 
made in the update (Table 2, Figure 2). These hand- 
books recommended actions to identify the main 
changes in the update without any difficulty, including 
a table of updated evidence, summary reports, or high- 
light the updated parts in the text with a red font 
[5,32,33,37,47]. 

Three handbooks (8.6%) provide guidance on how to 
publish and disseminate the updated CPG. All three of 
them include methods to disseminate the updated CPG 
as widely as possible by publishing in relevant indexed 
journals [5], disseminate within the patient organization 
of the specific disease [48], or working together with 



Identification of relevant new evidence: 

• ""All comments received on published SIGN guidelines, or Information on Important 
new evidence In ttie field, or evidence of impacts on equality groups are fed back to 
tfie guideline development group, either for Immediate response or for more 
detailed consideration on review of the guideline."^'* 

• "The editorial board meets once a month, and at every meeting, one speciality or a 
group of topics are discussed with 1-3 top experts on the field invited to attend. "^^ 

Assessment of the need for an update: 

• "The editorial team produces and updates evidence summaries continuously, and 
whenever the evidence summaries give rise to updates to the guidelines, the 
guidelines are updated. "^^ 

• "At this point the group should determine the extent of the update required. In 
addition, the composition of the group should be reassessed based on the planned 
extent of the update. "^ 

Updating process: 

• "An update search is carried out looking for evidence based guidelines, HTAs, and 
systematic reviews produced since publication of the last version of a guideline. 
These searches are based on the key questions and search strategies used In the 
original guidelines. 

• Use of critical appraisal files like: OstFLCrftIca, the free-access critical appraisal 
files IT application of Osteba. "^ 

Edition and dissemination: 

• "In EBfJI Guidelines, updated content appears in red font for 6 months after the 
update was made, "^^ 

• "Publish the changes using different methods: publishing in relevant indexed 
journals and/or the journals of the societies Involved, indexing In their own website 
or In other international sources like the National Guideline Clearinghouse (NGC). 



Figure 2 Box of relevant comments. 



Vernooij ef al. Implementation Science 2014, 9:3 
http://www.implementationscience.eom/content/9/1/3 



Page 6 of 9 



public and private partners to reach specific groups and 
individuals [43]. 

Discussion 

We systematically reviewed 35 methodological hand- 
books that provide some type of guidance on the updat- 
ing process of CPGs. Our results show that overall the 
updating guidance is poorly described. Crucial elements 
in identifying new evidence, the assessment for the need 
for an update and the updating strategy itself, are gener- 
ally lacking or include solely a reference to the deve- 
lopment process. Our findings resonate with previous 
findings that suggest that there is a need for rigorous 
international guidance for updating CPGs [8,14]. 

Figure 3 summarizes an updating process framework 
for CPGs based on a previous systematic review from 
our group and the results of the present study [8]. The 
process of updating a CPG starts with assembling a 
group responsible for updating the CPG. However, we 
found that the majority of the institutions (65.7%) do 
not include any information about this first step. There 
is no clear consensus on who should participate in an 
updating process and, consequently different organiza- 
tions use different strategies, depending on the charac- 
teristics of the organization and type of update. An 
updating working group, should consist of individuals 
with a background in methodology and experts in the 
field of interest, just as the original guideline group [5]. 



New developments in the clinical area, such as new 
technologies, might require including additional mem- 
bers with different expertise. 

The actual updating process starts with identifying 
new relevant evidence. Currently, the period between 
the last publication of the CPG and starting the updating 
process (time frame) is frequently determined at the 
time of publication. The majority of the handbooks 
(62.9%) include a fixed time frame from two to five 
years, consistent with the results of previous research by 
Shekelle et al. [13]. This study including a sample of 17 
guidelines, estimated that approximately one-half of the 
CPGs will be outdated after 5.8 years (95% CI: 5.0 - 6.6), 
and 10% are obsolete after 3.6 years (95% CI: 2.6 - 4.6) 
[13]. However, these average estimates can be misleading 
as CPG deteriorating speed is highly topic-specific, with 
some fields with rapid developments requiring more 
frequent surveillance for new evidence than others. 
Suboptimal time frames are likely to result in guidelines 
becoming obsolete or inefficient use of resources. 

After identifying new relevant evidence, an assessment 
of the effect of this new evidence should be conducted, 
determining the need for an update [5,9-11]. We believe 
that this process is best conceptualized as a two-stage 
process because these are two independent stages with 
identifying possible new relevant evidence as first step, 
and, subsequently, deciding whether the identified evi- 
dence this evidence alters the validity of the current 



Assemble the group responsiblefor updating the CPG: Similarities with the group responsible for developing 
the CPG with multidisciplinary contributions. The profiles required vary in the topic and type of update. 



IE 



1: Identify new relevant evidence 



Different methods for identifying new relevant evidence; Literature search, restrictive or exhaustive; 
collecting opinions of experts; external review; or alerts. 

The time frame for assessing the need of an update differs and ranges from 6 months to 5 years. 



IE 



2: Assessment of the need for an update 



Different methods for assessing the need for an update; Assessment of importance and relevance of new 
evidence, whether new evidence alters current CPG, whethernew evidence is not yet included; group 
meetings; expert judgement; or producing continuously evidence summaries. 



No effect: validity remains 
No update required 



Obsolete CPG 
Update required 



IE 



3: Updating process 



Literature search, evidence selection, evidence synthesis, evidence assessment 

Methods might be similar to step 1 and 2, consequently, the results might be used whenever this has been 
done rigorously. 



IE 



4: External review 



A broad-ranging external review with a multidisciplinary sample, not involved in developing or updating the 
CPG, should be conducted, including experts in the clinical area and methodological experts 



IE 



5: Publication 



Publication: The publication includes an overview of updated recommendations in each chapter, additionally, 
c/e novo recommendations ornew fields should be indicated. 



Figure 3 The updating process of CPGs. 
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recommendations as second step. However, at the 
moment, formal explicit procedures for assessing the 
need for an update are not available, with most of the 
included handbooks (77.1%) not providing explicit 
methods for assessing the need for an update. 

When the need for an update is confirmed, the new 
evidence has to be incorporated in the current recom- 
mendations. However, less than one-half of the included 
handbooks state specific methods for this process. Previ- 
ous studies suggest a model of assessing the need for an 
update using expert opinion, focused literature reviews, 
and consensus meeting [11,13]. A reference to the devel- 
opment process, often included in the evaluated hand- 
books, is not enough because the aim of any update 
should be to incorporate new evidence in the context 
of previous recommendations. More specific methods 
should be included in the handbooks. 

A further problem is that several institutions use dif- 
ferent terminology and consequently bring further con- 
fusion. Some institutions use the term 'monitoring' for 
the identification of new evidence and assessment of the 
need for an update, often within an abridged time frame 
[5,14,32,33,37,43,44,52]. In addition, the term 'dynamic 
updating' and 'living guideline' is used indistinctively, 
suggesting that CPGs are updated promptly and are 
always up-to-date [14,40,46]. Nevertheless, none of these 
handbooks provide guidance for conducting these pro- 
cesses and there is no consensus on when a guideline 
starts being dynamic or can be considered as a living 
guideline (Figure 3). We suggest avoiding these terms 
because it solely reflects the aspect of time between two 
versions. In Figure 3, we include a proposal regarding 
consistent terminology. Further research and consensus 
is needed in the international community about coher- 
ent terminology. 

Our study is, as far as we know, the first study to 
examine the guidance about the updating process pro- 
vided by CPG methodological handbooks. Our work has 
several strengths. We conducted a systematic and ex- 
haustive search that included main databases, clearing- 
houses, and several institutions identified by a previous 
survey [12]. In addition, we contacted several organiza- 
tions to retrieve non-published handbooks; therefore we 
believe that we included most of the existing handbooks. 
We independently performed eligibility and data extrac- 
tion with a CRF developed and piloted by a group with 
extensive experience in the field. 

Our study, however, might be subject to some limita- 
tions. It is possible that, after our extensive literature 
search, we did not identify all available handbooks because 
some are not indexed nor published, and only used for in- 
house purposes. However, unpublished handbooks are 
likely to be of lower quality. If this is the case, it would 
imply that we overestimated the quality of the updating 



guidance, further strengthening our conclusions. Finally, 
the reported methods in handbooks might not reflect the 
actual updating in CPGs. However, we believe that this is 
unlikely given previous results of our international survey 
with CPG developers [12]. 

Conclusion 

Our work shows that updating guidance included in 
CPGs methodological handbooks is overall of poor qual- 
ity. CPGs developers should provide more explicit and 
rigorous guidance and standardize the terminology used. 
This could, consequently, lead to a more optimal updat- 
ing process of CPGs, and ultimately, to valid trustworthy 
guidelines. 
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